Search for: All records

We present a simple approach to make pre-trained Vision Transformers (ViTs) interpretable for fine-grained analysis, aiming to identify and localize the traits that distinguish visually similar categories, such as bird species. Pre-trained ViTs, such as DINO, have demonstrated remarkable capabilities in extracting localized, discriminative features. However, saliency maps like Grad-CAM often fail to identify these traits, producing blurred, coarse heatmaps that highlight entire objects instead. We propose a novel approach, Prompt Class Attention Map (Prompt-CAM), to address this limitation. Prompt-CAM learns class-specific prompts for a pre-trained ViT and uses the corresponding outputs for classification. To correctly classify an image, the true-class prompt must attend to unique image patches not present in other classes' images (i.e., traits). As a result, the true class's multi-head attention maps reveal traits and their locations. Implementation-wise, Prompt-CAM is almost a "free lunch," requiring only a modification to the prediction head of Visual Prompt Tuning (VPT). This makes Prompt-CAM easy to train and apply, in stark contrast to other interpretable methods that require designing specific models and training processes. Extensive empirical studies on a dozen datasets from various domains (e.g., birds, fishes, insects, fungi, flowers, food, and cars) validate the superior interpretation capability of Prompt-CAM. The source code and demo are available at https://github.com/Imageomics/Prompt_CAM. 

We investigate the quantum dynamics of a spin coupling to a bath of independent spins via the dissipaton equation of motion (DEOM) approach. The bath, characterized by a continuous spectral density function, is composed of spins that are independent level systems described by the su(2) Lie algebra, representing an environment with a large magnitude of anharmonicity. Based on the previous work by Suarez and Silbey [J. Chem. Phys. 95, 9115 (1991)] and by Makri [J. Chem. Phys. 111, 6164 (1999)] that the spin bath can be mapped to a Gaussian environment under its linear response limit, we use the time-domain Prony fitting decomposition scheme to the bare–bath time correlation function (TCF) given by the bosonic fluctuation–dissipation theorem to generate the exponential decay basis (or pseudo modes) for DEOM construction. The accuracy and efficiency of this strategy have been explored by a variety of numerical results. We envision that this work provides new insights into extending the hierarchical equations of motion and DEOM approach to certain types of anharmonic environments with arbitrary TCF or spectral density.